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^, Abstract. The paper is devoted to interrelation between the zeta 

distribution Z{f, a — n) = J f{x) det(x'a;)^"~"-'/^da; and the Radon 
\f~^ \ transform on the space ^n,m oi n x m real matrices x = {xi,j). 

We present a self-contained proof of the Fourier transform formula 
for this distribution. Our method differs from that of J. Faraut 
.^ ■ and A. Koranyi [FK] in the part related to justification of the 

fy I corresponding Bernstein identity. We suggest a new proof of this 

identity based on explicit representation of the radial part of the 
Cayley-Laplace operator A = det{d'd), d = {dij)nxm- We also 
C^ ' study convolutions with normalized zeta distributions, and the cor- 

responding Riesz potentials. The results are applied to investiga- 
tion of Radon transforms on 97l„^m- 
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Diverse problems of integral geometry in the iV- dimensional real 
space M^ get a new flavor if the dimension A^ has the form A^ = nm, and 
M^ is identified with the space 2Jln,m of n x m real matrices x = (xj '' 
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2 BORIS RUBIN 

If we accept this point of view, a number of new "higher rank" phe- 
nomena will come into play. In the present paper we focus on analytic 
tools which allow to investigate the properly defined Radon transform 
f{x) -^ fij) on Mn,m- This transform assigns to a function /(x) on 
Mn^m a collection of integrals of / over the so-called matrix k-planes 
T in Mn,m- Each such plane is, in fact, an ordinary fcm-dimensional 
plane in M"'", but the manifold 1 of all matrix A;-planes is essentially 
"smaller" than the manifold of all A;m-dimensional planes in M"'". The 
main problem is to reconstruct f{x) from known data /(r), r G T, for 
possibly large class of functions /. 

In the rank-one case m = 1 this problem is well investigated; see, 
e.g., [E], [Hel], [Ru2], and references therein. The standard tools are 
the Fourier analysis, convolution operators, and the corresponding real- 
variable techniques, including approximation to the identity, singular 
integral operators, and fractional integrals. In the higher rank case 
771 > 2, all these tools still require a further development. 

Let us pass to details. Suppose that f{x) is a Schwartz function on 
^n,m; u > 771. The Fouricr transform of / is defined by 



(1-1) {^f){y) 



j exp{ti{iy'x))f{x)dx, y = {yij) e Tin,., 



Mr, 



where y' denotes the transpose of y, dx = HiLi YlT=i d^ij- This is the 
ordinary Fourier transform on M"'". Given a complex number a, we 
denote 



(1.2) Zif,a-7i)= J f{x)\xC-''dx 

971 

where \x\m = det(x'a;)^/^ is the volume of the parallelepiped spanned 
by the column- vectors of the matrix x. For m = 1, \x\m is the usual 
euclidean norm on M". Integrals like ()1.2j) are known as zeta i7itegrals, 
the ^'-distribution defined by analytic continuation of (jl.2|) is called a 
zeta distributio7i, and the corresponding function a — > Z{f, a — ti) is 
sometimes called a zeta function. 

We begin our study by giving a relatively simple self-contained proof 
of the well-known functional relation 



r^(«/2) r^((n-a)/2)' 

where the normalizing denominators are Siegel gamma functions asso- 
ciated to the cone of positive definite m x m matrices. Both sides of 
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(jl.Hj) are understood in the sense of analytic continuation. The equal- 
ity (jl.3p is of fundamental importance for the sequel. It is, in fact, a 
Fourier transform formula for the zeta distribution. Convolutions with 
zeta distributions coincide (up to normalization) with Riesz potentials 
on 9K„ m playing a vital role in integral geometry and analysis on matrix 
spaces [ORl], [P], [Sh]. 

Our interest to this topic is also motivated by the following. A 
number of proofs of the equality (|1.3p and its generalizations can be 
found in the literature; see, e.g., [CI], [Far], [FK], [Ge], [Ra], [Sh]. 
Unfortunately, these proofs are not "self-contained enough" and some 
important technical details are skipped. These details are crucial by 
taking into account an essential difference between the case 2m < n + 2 
and 2m > n + 2. This difference is not indicated in some papers. The 
crux is that the distribution on the left-hand side of ()1.3p is regular 
if and only if Re a > m — 1, whereas the right-hand side is regular if 
and only if Re a < n — m + 1. For 2m > n + 2 these two sets on the 
complex plane are separated one from another! In the case 2m, < n + 2 
the proof is elementary [ORl]. For 2m > n + 2, justification of (jl.3j) 
represents a difficult problem. 

The phenomenon of lack of common domain of regularity was inves- 
tigated by E. Stein [Stl] for Riesz distributions on the space of complex 
n X n matrices. For n = m, the formula ()1.3|) includes the result of 
S. S. Gelbart |Ge| Chapter IV] and a special case of Proposition II-9 
of M. Rais |Raj . For n > m, this formula can be found in |Farj . |FK| 
Theorem XVI, 4.3], and jOj in a more general context of representa- 
tions of Jordan algebras; see also [Sh, p. 108]. For more information 
on zeta distributions, see [Ig], [SS], [Shin], and references therein. 

Our proof of (jl.3|) utilizes the idea from jFKj to employ /C-Bessel 
functions and the so-called Bernstein identity 

(1.4) A\x\t = B{X)\x\i-', 

m—l 

13{X) = (-1)™ Y[{X + t){2-n-X + t), 

i=0 

where A is the Cayley-Laplace operator defined by A = det{d'd), d = 
{di,j)nxm- This identity amounts to pioneering papers of J. Bernstein; 
see, e.g., [B], [Ig]. A novelty of our approach is that we first derive an 
explicit formula for the radial part of A, and then use it for justification 
of ()1.4|) . It is worth noting that the proof of the Bernstein identity in 
|FKj and jOO employs the heat kernel and integration by parts over 
the corresponding cone. Justification of this integration by parts was 
unfortunately skipped. That was one of the reasons why we tried to 
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find an alternative proof. We liope tliat our metliod can be extended 
to more general settings similar to tliose in |FKj and [Oil- 

Tlie second concern of the paper is the so-called Wallach set W of 
the normalized zeta distribution 



(1.5) (Ca,/)=Q-C. / /(X)|x|^"rfx, feS{^n,m): 

im(a/^) J 
an 

where "a. c." abbreviates analytic continuation. This is an entire func- 
tion of a. The set W consists of two components, one of which is 
continuous and another is discrete. The continuous component {a : 
Rea > m — 1} is constituted by those a G C for which the distribu- 
tion ()1.5p is regular, that is, the integral in ()1.5|) absolutely converges. 
The discrete component consists of non-negative integers a = k, k = 
0, 1, . . . , m — 1 (outside of the domain of regularity), for which the dis- 
tribution ()1.5|) is a positive measure. We obtain explicit representation 
of (a for all a G W, and investigate convolutions (Ca * /) (x) assuming 
/ G L^{D}ln^m)- In particular, under some natural restrictions on n and 
m, we prove that these convolutions are well defined almost everywhere 
on dJln,m provided 

71 

(1-6) l<P<-f, r. 

Ke a + m — 1 

This result agrees with the case m = 1 in [St2]. The question whether 
the condition ()1.6p is sharp remains open. 

Another normalization of the integral p.2|) gives rise to the Riesz 
potential 



(1.7) (/"/)(x) = / fix-y)\yC--dy, 

7n,m(tt) J 

2-7r-/^r^(a/2) 
r„((n-a)/2) 

the Fourier transform of which is \y\^{^f){y) in a suitable sense. The 
operator /" can be regarded as the (— a/2)th power of (— 1)'"A where 
A is the Cayley-Laplace operator. 

Our investigation of convolutions {Ca* f){x) and the Riesz potentials 
/"/ employs matrix modification of the Gauss- Weierstrass integral (see 
[SW], [Ta] for m = 1) defined by 



;i.8) iWtf){x)= J htix-y)f{y)dy, 



Tin 
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where t is a positive definite m x m matrix, and 

(1.9) ht{x) = (47r)-"'"/2 det(t)-"/2 exp(-tr(t-^x'x)/4) 

is the relevant heat kernel. This approach allows to avoid essential tech- 
nical difficulties. The idea is to represent the Riesz potential through 
the lower-dimensional Garding-Gindikin fractional integral by a simple 
formula 

(1.10) (Wtirmx) = {d'g.m, gM = {Wtf){x), 

provided m — 1 < Re a < n — m + 1. We recall that the Garding- 
Gindikin fractional integral on the cone Vm of positive definite matrices 
has the form 

oo 

(1.11) {I'_gm = YJx)j 9{T)det{T-tf-'dr, 

t 

where (i = (m + l)/2, Re \> d — 1, and integration is performed over 
all r G Vm so that t — t E Vm ■ These integrals were introduced by 
Lars Garding |Galj who wrote |Ga2j : 

"... Actually the origin of my integral was a statistics paper (samples 
of mean, variance etc. from a multivariate Gaussian distribution). The 
idea to use it in analysis came from my many meetings with my beloved 
teacher Marcel Riesz". 

Integrals (jl.llj) were substantially generalized by S. Gindikin [Gij . 
They have proved to be an important tool in PDE |VGj . |Rabj . and in 
the theory of Radon transforms on Grassmann manifolds and matrix 
spaces |GRj . [0R2], |Ru3j . If t = 0, then Wt turns into the identity 
operator and p.lO|) reads 

(1.12) (r/)(x) = -— ^ / W-\Wtf){x)dt. 

Vm[a/2) J 

Vm 

Our application of zeta distributions to integral geometry is based on 
intimate connection between the Riesz potentials (jl.7|) and the matrix 
fc-plane Radon transform / — >■ / on 97t„,m- This connection is realized 
via the generalized Fuglede formula 

(1.13) (/y(a;) = const (/V)(x) 

which is well known for m = 1 [Fu], [Hel]. Here the left-hand side is 
the mean value of /(r) over all matrix fc-planes r "passing through 
x" . For sufficiently good functions /, this formula was established in 
[ORl]. We justify it for all / G LP(97t„,^), 1 < p < n/{k + m - 1), 
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and apply to the inversion problem for the Radon transform f ^ f. 
Solution to this problem is presented in the framework of the relevant 
theory of distributions. For m = 1, such a theory was developed by 
V.I. Semyanistyi [Se]. 

The paper is organized as follows. In Section 2 we establish our no- 
tation and recall some basic facts important for the sequel. In Section 
3 we derive a formula for the radial part of the Cayley-Laplace operator 
and prove the Bernstein identity (jl.4|) . In Section 4 we study the zeta 
integral (|1.2p . prove (|1.3|) . and obtain explicit representations of the 
normalized zeta distribution ()1.5|) in the discrete part of the Wallach 
set. Convolutions with zeta distributions and Riesz potentials of L^ 
functions are investigated in Section 5. Section 6 is devoted to appli- 
cations of the results of preceding sections to integral geometry. We 
define the matrix fc-plane Radon transform and justify the generalized 
Fuglede formula ^TT^ for / e LP{mn,m), 1 < P < n/{k + m - 1). 
Then we introduce the space $' of distributions of the Semyanistyi 
type. Two ^'-distributions coincide if and only if their Fourier trans- 
forms differ by a tempered distribution supported by the singular set 
{y : y & OT„,m,i'ank(t/) < m}. An inversion formula for the Radon 
transform / — * / for / G L^(OT„m) is then obtained in the $'-sense. 
For convenience of the reader, evaluation of some useful integrals is 
presented in Appendix. 

The paper contains a number of open problems that might be of 
interest. These are stated in Remarks 15.5115.111 and 16.91 

Acknowledgements. I am grateful to Prof. Jacques Faraut for 
valuable remarks, and Prof. Jean-Louis Clerc for correspondence. Spe- 
cial thanks go to Prof. Evgenyi Evgenyievich Petrov whose pioneering 
paper [P] and friendly comments were so inspiring. I am also indebted 
to Dr. Sergei Khekalo and Dr. Elena Ournycheva for useful discussions. 



2. Preliminaries 

2.1. Matrix spaces. Let Tln,m be the space of real matrices having 
n rows and m columns. We identify Mn^m with the real Euclidean 
space M"™. The letters x, y, r, s, etc. stand for both the matrices 
and points since it is always clear from the context which is meant. If 
X = (xij) G Mn,m, we write dx = YYi=i YYjLi ^^i,j ^^ ^^^ elementary 
volume in Wln,m- In the following x' denotes the transpose of x, Im is 
the identity m x m matrix, stands for zero entries. Given a square 
matrix a, we denote by |a| the absolute value of the determinant of a; 
tr(a) stands for the trace of a. 
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Let Sm be the space oimxm real symmetric matrices s 



'"l;3J^ "^*J 



Sj^i. It is a measure space isomorphic to M™'(™-+i)/2 with the volume el- 
ement ds = Y[i<j dsij. We denote by Vm the open convex cone of 
positive definite matrices in Sm] Vm is the closure of Vm that consists 
of positive semi-definite matrices. For r G Vm we write r > 0. The 
inequality ri > r2 means ri — r2 G Vm- If O' and h are positive semi- 
definite matrices, the symbol / f{s)ds denotes integration over the 
set 

{s : s G Vm, a < s < b} = {s : s — a e Vm, b — s E Vm}- 

The group G = GL^m, M) of real non-singular mxm matrices g acts on 
Vm transitively by the rule r -^ g'rg. The corresponding G-invariant 
measure on Vm is 



(2.1) 4r = |r|"''c?r, |r| = det(r), d={m + l)/2 

|Te| p. 18]. The cone Vm is a G-orbit in Sm of the identity matrix Im- 
The boundary dVm of Vm is a union of G-orbits oi m x m matrices 



ek 



Ik 




k = OA, 



,m — 1. 



More information about the boundary structure of Vm can be found in 
jEH P- 72], and ^m\ P- 78]. 
We denote by Tm a group of upper triangular matrices 



(2.2) 



t 



1,1 







ti.i > 0, 



Each r G Vm has a unique representation r = t't, t G Tm, so that 



f{r)dr 



Pn 



(2.3) 



^™l '^^1,1 / '^2^2 '^^2,2 • • • 




Jy^l,!, - - - , i^m,m) 



f{t't) dU, dU 



^m(m — l)/2 



Yidt 

i<j 



*J' 
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[T^ p. 22] , |Mu| p. 592]. In the last integration, the diagonal entries 
of the matrix t are given by the arguments of /, and the strictly upper 
triangular entries of t are variables of integration. 
Let us recall some useful formulas for Jacobians. 

Lemma 2.1. (see, e.g., |Mu| pp. 57-59]). 

(i) If X = ayb, where y G 93t„,m, a G GL{n,M.), b G GL^rrijE.), then 

dx=\a\"'\b\''dy. 

(ii) If r = q'sq, where s G Sm-, Q G GL{m,M.), then dr = \q\"^^^ds. 

{iii) If r = s^^, where s G Vm, then r G Vm, and dr = \s\^"^^^ds. 

In the following (A, m) = A(A + 1) . . . (A + ?ti — 1) is the Pochhammer 
symbol, 6ij is the Kronecker delta; ^^a.c." abbreviates analytic contin- 
uation. To facilitate presentation, we shall use the symbols " ~ " and 
" < " , instead of "=" and "<" , respectively, to indicate that the corre- 
sponding relation holds up to a constant multiple. All standard spaces 
of functions of matrix argument, say, x = {xij), are identified with the 
corresponding spaces of functions of nm variables xi_i,xi,2, • • • ,Xn,m- 
The Fourier transform of a function / G L^(Mn,m) is defined by 

(2.4) iJ^f)iy)= / exp{tr{ty'x))f{x)dx, y E mn,m. 

2.2. Gamma functions, beta functions, and Bernstein polyno- 
mials. The Siegel gamma function associated to the cone Vm is defined 
by 

(2.5) r^(a) = / exp(-tr(r))|r|"-'^dr, d = {m + l)/2. 

Vm 

It is easy to check |Mut p. 62], that the integral (j2.5j) converges 
absolutely if and only if Re a > d — 1, and represents a product of 
ordinary gamma functions: 

m— 1 

(2.6) r^(a) = vr"^(— 1)/4 J] r(« - j/2) . 

j=0 

This implies 

(2.7) {-ir Yl~tf = 2-™£(^ . 2-™(„. „). 

r„(-a/2) r(a) 

(a, m) = a{a + 1) ■ ■ ■ {a + m — 1) being the Pochhammer symbol. If 
1 < k < m, k eN, then 

(2.8) Tm{a) = 7r''^^-''^/^Tkia)Tm-kia - k/2), 
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.2 n) r^(«) ^ Tkia + ik-m)/2) 

^ ■ ' TUa + k/2) Tkia + k/2) ' 

The beta function of the cone Vm is defined by 



(2.10) Bm{a,p)= f \r\''-'^\Im~rf~'^dr, d={m + l)/2. 



This integral converges absolutely if and only ii Rea,Re P > d — 1, 
and obeys the classical relation |FKt p. 130] 

(2-11) Bm{a,/3) = / , m • 

Let r = (rjj) G Pm- We define the following differential operator 
acting in the r-variable: 

(2.12) D.Z), = det(,„,^). ■-,., = { 1/2 j^^J^ 

Lemma 2.2. 

(i) For r G Vm o-nd z G 5^ (the complexification of Sm), 

(2.13) Djexp(-tr(r2))] = (-l)"det(z) exp(-tr(r2)). 
(ii) For r G "Pm o'^c? a G C, 

(2.14) D{\r\^"'/Vm{a)) = \r\''-^-'' /Vm{a - 1), rf = (m + l)/2. 

The proof of this statement is simple and can be found in [Gall p. 
813], |FK1 p. 125]. 

By changing notation and using (j2.7|) . one can write (j2.14|) as 

(2.15) F)|r|" = 6(a)|r|"-\ 
where 

(2.16) 6(a) =a(a + 1/2) ■••(« + t/-l) 

is the so-called Bernstein polynomial of the determinant JFKJ. 



2.3. Bessel functions of matrix argument. We recall some facts 
from jHerzj and |FKj . 
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2.3.1. J -Bessel functions. The JT'-Bessel function Ju{r), r G "Pm, can 
be defined in terms of the Laplace transform by the property 

(2.17) / e^^{-ii{zr))J^{r)\rf~'^dr = r^(z/) exp(-tr(^~l))det(z)~^ 

d= {m + l)/2, ze%n = Vm + iSm 

(the branch of the multi- valued function det(z)~'^ is chosen so that for 
z G Vm, the argument of det(z)~'^ is zero). This gives 

(2.18) X(r)|rr-^ = ^^ J exp{tT{rz - z-'))det{z)-'dz, 

Re z=cro 

(To € Vm, N = m{m + l)/2, or, by changing variable, 

(2.19) J,(r) = ^^ [ expiti{z~rz-'))det{z)-^dz. 

{2my'' J 

Re z=ao 

Both integrals are absolutely convergent for Re v > m. Applying the 
operator ()2.12j) . we obtain the following recurrence formulae: 

(2.20) D[Mr)\rr'^] = ^;i^^X-l(r)|rr-l-^ 

(2.21) D[X{r)] = (-ir p^r^^\, X+i(r). 

The equation ()2.2()j) allows to extend Jlu{r) analytically to all u & C. 

There is an intimate connection between the Fourier transform and 
i7-Bessel functions. 

Theorem 2.3. jHerzl p. 492], |FK| p. 355] Let f{x) be an integrable 
function on 9Jl„,m of the form f{x) = /o(x'x), where /q is a function 
on Vm- Then 

(2.22) (.F/)(y)= / exp{tT{zy'x))fo{x'x)dx = --—fo(^ 

J Tm{n/2) V 4 

where 

(2.23) Ms) = J Xi^{ts) |rr/2-'^/o(r)rfr, rf = (m + l)/2, 



is the Hankel transform of /( 



0- 
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This statement is a matrix generalization of the classical result of S. 
Bochner, see, e.g., |SWj . Chapter IV, Theorem 3.3. 

One can meet another notation for the iJ-Bessel function in the 
literature. We follow the notation from |FKj . It relates to the notation 
As{r) in |Herzj by the formula Jlu{f^) = rm(i^)^<5(^), S = u — d. For 
m = 1, the classical Bessel function Ju{r) expresses through Jlu{r) as 

2.3.2. IC-Bessel functions. Let, as above, r G Vm, d = {m + l)/2. The 
/C-Bessel function IC^ir) is defined by 

(2.24) /C^(r) = I exp(tr(-s - rs-^))\s\'''^ds, 
or (replace s by s^^) 

(2.25) lC^{r) = f exp(tr(-s-^ - rs))\s\-''-'^ds, 

I m 

see IE], WM- For m = 1, 

oo 

(2.26) /C^(r) = f exp{-s - r/s)s''-^ds = 2r'^/2^^(2v^), 



Ki, being the classical Macdonald function. 

Lemma 2.4. Let r G V^, d = {m + l)/2. 

(i) The integral \2. 24\j (or \2.2^) ) converges absolutely for all z/ G C, 

and is an entire function of v. 

(ii) The following estimates hold: 

(a) For Rev > d-l: 

(2.27) |/C,(r)|<r^(i?ez/); 
(h) For Reu < 1 - d: 

(2.28) |/C,(r)|<r„(-i?ez/)|r|«^^; 
(c) Fori- d< Reu < d-l: 

(2.29) |/C,(r)| <r^(rf) + |r|^-''-T^(ci-l + £), Ve > 0. 
(iii) If Reu < 1 — d, then 

(2.30) lime-'^''}C^{er) = r^{-u)\r\\ 
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Proof. All statements, with probable exception of ()2.29|) . are known 
jEEl- The estimate fITTTI) follows from (IT^ : (IT^ and ^^^ are 
consequences of fl2.25|) . To prove ()2.29|) . we write 



JC.ir) = 


f + f exp(tr(-s-rs-^)) sr-'^rfs = 


h + h. 


\\s\>l \s\<l J 




For Ji we have 




\h < 1 exp(tr(-s))c?s < T^{d) 
J 




\s\>l 




provided Reu < d. For J2, by changing variable s -^ s^^, 


we obtain 




l2= [ exp(tr(-s-^ - rs)) s -""^ds. 





|s|>l 

If i?ez/ > 1-rf, then \s\-^^^-'^ < \s\^-^ We > 0, and 



< / exp(tr(— s "^— rs))|s|^ ^ds 



\s\>l 



< / exp(tr(— rs))|s|^ ds 

= ^„(d-l + £)|r|l-'^-^ 

This gives fTI^ . D 

2.4. Stiefel manifolds. For n > m, let Vn^m = {f G Tln.m '■ v'v = Im} 
be the Stiefel manifold of orthonormal m-frames in M". If ra = m, 
then Vn^n = 0{n) is the orthogonal group in R". The group 0{n) acts 
on Vn^m transitively by the rule g : v ^ gv, g G 0{n), in the sense 
of matrix multiplication. The same is true for the special orthogonal 
group SO{n) provided n > m. We fix the corresponding invariant 
measure dv on Vn,m normalized by 

(2.31) (Jn,m= / dV- 



rm{n/2y 

Lemma 2.5. |Mu| p. 589] Let x and y be real matrices such that x is 
nx k and yismxk,n>m. Then x'x = y'y if and only if there exists 
V G Vn^m such that X = vy. In particular, if x is n x m, n > m, then 
there exists v G Vn^m such that x = vy, where y = {x'xY^'^. 
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Lemma 2.6. (polar decomposition). Let x G Tln,m, n > m. If 
rank(x) = m, then 

X = OT^/^, V e Vn,m, r = X'x e Vm, 

and dx = 2-™|r|("-™-i)/2rfrrft;. 

This statement and its generalizations can be found in different 
places, see, e.g., [Tfa^ p. 482], [H PP- 66, 591], |FTI p. 130]. 
A modification of Lemma 12.61 in terms of upper triangular matrices 

t G Tm (see ()2.2|) ) reads as follows. 

Lemma 2.7. Let x G Tln,m, n > m. If rank(x) = m, then 

X = vt, V e Vn,m, t G Tm, 

so that 

m 

dx = Y^t'Jj-' dtjjdt:fdv, dt^, = Y\dtij. 

j=l i<j 

Proof. This statement is also well known. It can be easily derived from 
Lemma IT^ and ()2.3p . Indeed, if rank(x) = m, then x'x G Vm and there 
exists t G Tm such that x'x = t't. We set v = xt~^ . Then v'v = Im 
and, therefore, v G Ki,m- This proves the representation x = vt, where 
V G Vn^m and t G T^. Furthermore, by Lemma ITHl and (j2.3p . 

fix)dx = 2-"" f dv f \r\^''-"'-^^/^f{vr^/^)dr 

VJln,m yn,m Im 



dv / f{v{t'ty/^)Ylt]]^ dtjjdtjv . 

Now we denote A = (t't^^'^t^^ G 0{m), then change the order of 
integration, and set vX = u. This gives 

/m „ 

W^'jj^'^^jd'^^* / f{ut)du, 

and we are done. D 

3. Radial functions and the Cayley-Laplace operator 

A function f{x) on 9Jln,m is called radial, if there exists a function 
/o(r) on Vm such that f{x) = fo{x'x) for all (or almost all) matrices 
X G ^n,m- One can readily check that / is radial if and only if it is 
0{n) left-invariant, i.e., f{'jx) = f{x) for all 7 G 0{n). 
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The Cayley- Laplace operator A on the space dRn,m of matrices x = 
(xij) is defined by 

(3.1) A = det(9'9). 

Here 9 is an n x m matrix whose entries are partial derivatives d/dxij. 
In the Fourier transform terms, the action of A represents a muhiph- 
cation by the polynomial (— l)™P(y), where y = (yij) G Mn^m, 



P{.y) = \y'y\= det 



yi-yi ■ ■ yi-y-n 



ym ' yi ■ ■ ym ■ ym 



yi, . . .ym are column-vectors of the matrix y, and " ■ " stands for the 
usual inner product in M". Clearly, P{y) is a homogeneous polynomial 
of degree 2m of nm variables y^.j, and A is a homogeneous differential 
operator of order 2m. For m = 1, it coincides with the Laplace operator 
onR'^. 

The Cayley-Laplace operator ()3.1|) and its generalizations were stud- 
ied by S.P. Khekalo |Khj . For m > 1, the operator A is not elliptic 
because P{y) = for all non-zero matrices y of rank < m. Moreover, 
A is not hyperbolic, although, for some n, m and i, its power A^ enjoys 
the strengthened Huygens' principle; see |Khj for details. 

Our nearest goal is to find a radial part of A corresponding to the 
polar decomposition x = vr^^'^, v G Vn^m-, ^ = a;'x G Vm- For m = 1, 
the classical result states that the radial part of the Laplace operator 
on W is 



a-n ^ n-l ^ 



op op 
By changing variable r = p"^, we get 

(3.2) L = 4ri-"/2^r"/2|_. 

or or 

The following statement is one of the main results of the paper. It 
extends ()3.2|) to the higher rank case. 

Theorem 3.1. Let Vl C Tln,m b^ o?^ open set consisting of matrices of 
rank m; n > m > 1. If f{x) = fo{x'x), fo{r) G C^™'('Pm); then for 
X & Q, 

(3.3) (A/)(x) = (L/o)(x'x), 
where 

(3.4) L = 4"^|^|'^-"/2£)|^|n/2-d+i^^ d = {m + 1)/2, 
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D = det[((l + 5ij)/2)d/drij] being the operator WT^) . 

Proof. For m = 1, ()3.4j) coincides with ()3.2|) . To prove the theorem, 
we first note that without loss of generahty, one can assume / to be 
compactly supported away from the surface {x : det(x'x) = 0}. Oth- 
erwise, /(x) can be replaced by /i(x) = ip{x)ip{x)f{x), where ip and ifj 
are radial cut-off functions of the form 

ip{x) = ipQ{ii{x'x)), ?/'(x) = V'o(det(x'x)), v^o, V'o e C°°(]R+), 

, . / 1, if < p < AT, / 0, if < p < £, 

The positive numbers N and e should be chosen sufficiently large and 
small, respectively. 

By the generalized Bochner formula ()2.22|1 . 

where 

/i(s) = (-4)™y" |rs|j;,/2(rs)|rr/2-'^-Vo(r)t/r. 

Let us transform |rs| Jn/2(rs). By (jT^ and (J22II), 

= (-l)"^Jn/2(r)|rr/2-'^. 

Hence, 

\r\Jn/2{r) = (-l)"(LJ-„/2)(r), L = {\r\'+^--/'D\r\-/'''){\r\D). 

Since |r|°'+i~"/2Z}|r|"/^~'^ and |r|D are invariant differential operators 
with respect to the transformation r — > grg'., g G (^^(m, R), |FK1 p. 
294], then so is L. Thus for any s G Vm, 

\rs\Jr./2{rs) = (-l)"^(LX/2)(sr) = (-l)'"L,[X/2(sr)], 

where L,, stands for the operator L acting in the r-variable (here we 
use the symmetry property Ju{rs) = Ju{s^^'^rs^^'^)) . It follows that 

h{s) = A^ j Lr[Jn/2isr)]\rr/^-''-'Ur)dr 



A"^ J {Dr\rr /'-''+' Dr)[Jn/2{sr)]fo{r)dr. 



v„ 
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Owing to remark at the beginning of the proof, one can integrate by 
parts and get 

h{s) = {-4r j Dr[Jn/2{sr)]\rr/'-'^\Dfo){r)dr 
Jn/2{sr)\rr/'-'{Lfo){r)dr, 

Vm 

where L = 4rn^^^d-n/2^^^n/2-d+l-Q^ rjj^^g 

nm/2 r / 1 \ 

which imphes flH.Hp . D 

Example 3.2. Let /(x) = |a;|^, \x\m = det(x'x)"^/^. By Theorem 13.11 

and ()2.15|) . (A/)(x) = ({>{x'x), where 

= r'b{X/2)b{{n + X)/2~d)\r\^/^-\ 
Thus, we have arrived at the following identity of the Bernstein type 
(3.5) A\x\t = B{X)\xt-', 

where, owing to (j2.16|) . the polynomial B{X) has the form 



m—l 



(3.6) B{X) = {-irl[{X + t){2~n-X + t). 

i=0 

An obvious consequence of ()3.5|) in a slightly different notation reads 

(3.7) A'^lxC^^-^ = Bkia)\x\ 



ta—n 

\m 5 



m—l fc— 1 

(3.8) Bk{a) = Y[Y[{a-t + 2j){a-n + 2 + 2j+t) 

= Bk{n-a-2k). 
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4. Zeta integrals 
4.1. Definition and example. Let us consider the zeta integral 

(4.1) Z{f,a-n)= J f{x)\x\:,-''dx 

where /(x) is a Schwartz function on Mn^m, n > m, \x\m = det(x'x)^/^. 
The following example gives a flavor of basic properties of Z{f, a — n). 

Example 4.1. Let e{x) = exp(— tr(x'x)) be the Gaussian function. By 
Lemma 12.61 for i?e a > m — 1 we have 



(4.2) Z{e,a-n) = 2-'"(t„,^ nr|"/2-rfexp(-tr(r))rfr 

_nm/2 

d = (?7i + l)/2. On the other hand, the well known formula for the 
Fourier transform yields 

(4.3) iJ^e){y) = 7,--/'eiy/2), 
and for Re a<n — rra + lwe obtain 

nm/2 / \ \{n-a)/2-d 



Z{J^e,-a) = c„,™7r"™/" / |r|^"-"^/"-'^exp(-tr(r/4))dr 

(4.4) = <^r„((n-«)/2), rf„,„ = c„,„7r"'"/22-(n~a). 

Thus, after analytic continuation we obtain the following meromorphic 
functions: 

(4.5) Z{e,a -n) = Cn,ni^m{<y/'2), a 7^ m - l,m - 2, . . . , 

(4.6) Z{J^e,—a)=dn^m^m{{n—a)/2), a ^ n—m+l,n — m+2, . . . . 
These equalities imply the functional relation 



r™(a/2) r„((n-a)/2)' 

which is a prototype of similar formulas for much more general zeta 
functions. Since each side of ()4.7|) equals c„,m, this formula extends 
to all complex a. Note that excluded values of a in ()4.5|) and ()4.6p 
correspond to m > 2. If m, = 1, they proceed with step 2, namely, 
a 7^ 0, —2, —4, . . . , and a 7^ ra, ra + 2, . . . , respectively. 

In the following, throughout the paper, we assume m>2. 
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Lemma 4.2. Let f G S{DJln,m)- For Re a > m — 1, the integral \4-l^ 
is absolutely convergent, and for Re a < m — 1, it extends as a mero- 
morphic function of a with the only poles m — l,m — 2, . . . . These 
poles and their orders are exactly the same as of the gamma function 
rm{a/2). The normalized zeta integral Z{f, a — n)/rm,(a;/2) is an en- 
tire function of a . 

Proof. This statement is known; see, e.g., |Khj . [Sh]. We present the 
proof for the sake of completeness. The equahty ()4.5p says that the 
function a -^ 2{f, a — n) has poles at least at the same points and at 
least of the same order as the gamma function rm(«/2). Our aim is 
to show that no other poles occur, and the orders cannot exceed those 
of Tm{(x/2). Let us transform (j4.1|) by passing to upper triangular 
matrices t G Tm according to Lemma [2.71 We have 

» m 

(4.8) Z{f, a-n) = J F(ti,i, . . . , t^,„) J] t^pdt 



1,1 ) 

i=l 



F{ti^i,...,t^^rn) = / dt^ / f{vt)dv, dt^ = Y[dtij. 

IRm(m-l)/2 Vn,m *^^' 

Since F extends as an even Schwartz function in each argument, it can 
be written as 



F(ti^l, . . . , tm,m) — -^0(^1, 1; • • • ; t^. 



2 

mJ y 



where Fq G iS(M'") (use, e.g.. Lemma 5.4 from [Lrl p. 56]. Replacing 
tf j by Si^i, we represent (14. 8j) as a direct product of one- dimensional 
distributions 

m 

(4.9) Z{f, a-n)= 2--(n(sM)l""^"'^^', ^0(^1,1, • • • , s^,m)). 

i=l 

which is a meromorphic function of a with the poles m — l,m — 2,... , 
see jGShlj . These poles and their orders coincide with those of the 
gamma function rm(a/2), cf. ()4.5p . To normalize the function ()4.9|) . 
following [GShlj, we divide it by the product 

m ra—l 

n r((a - I + l)/2) = n r((« - 0/2) = r™(a/2)/7r"(™-i)/^ 

i=l i=0 

As a result we obtain an entire function. D 
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4.2. A functional equation for the zeta integral. Let us prove 
another main result of the paper. 

Theorem 4.3. If f E S{Ttn,m), n>m, then 

(4 10) 2{f,a-n) ^ ^-nml2c^m{c.-n) _ ^{0= f , -O) 



r„(a/2) r„((n-a)/2)' 

Proof. We recall that both sides of this equality are understood in the 
sense of analytic continuation. Furthermore, the distribution on the 
left-hand side of ()4.10|1 is regular if and only if Re a > m — 1, whereas 
the right-hand side is regular if and only if Re a < n — m + 1. For 
2m > n + 2 these two sets have no common points. We consider the 
cases 2m < n + 2 and 2m > n + 2 separately. 

(i) The case 2m. < n + 2. We start with the following equality: 

{Tf){y)\elr^ + y'yr/'dy 

(4.11) '^nm/2^m{n-a)/2 r / S \ 

Here /C(a_„)/2 is the /C-Bessel function (see Section 12. 3. 2|) . and 

(4.12) m-l<Rea<n-m + l 

(since 2m, < n + 2, the domain ()4.12|) is not void). Suppose for a 
moment, that ()4.11|) is true. Then the result follows if we pass to the 
limit as e — > 0. Indeed, by ()2.28|) for all e > we have 

(4. Id) \e " l^{a-n)/2y-^XXJ\<lm\ ^ I — 

Furthermore, by ()2.3()|1 . 

(4.14) lhn£-(--")/2/C(,_„)/2 (^-x'x) = 2™("-")r^ (^^) 






Hence, by the Lebesgue theorem on dominated convergence, we are 
done. 

To prove ()4.11|1 . let es{x) = exp(— tr(xsx')/47r), s G Vm- By the 
Plancherel formula, 

(4.15) |s|""/2 j (J^f)(y)exp{-tT{nys-'y'))dy= j f{x)es{x)dx. 
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Now we multiply KTE^ by |s|("-")/2-°'exp(-tr(£7rs"i)), d = (m + l)/2, 
then integrate in s, and change the order of integration. We obtain 



{^ f){y)a{y)dy = I f{x)b{x)dx, 

where 

a(y) = J\s\-"/^-''exp[~tTi7rs-\y'y + eIJ)]ds {s = r') 

= J |tr/2-'^exp[-tr(7rt(yV + elm))]dt 

and 

b{x) = /"|s|("-°)/2-'^exp[-tr(sa;V4vr + e7rs-^))]rfs 

= (£7r)'"("-")/2 /" |^|(n-«)/2-d exp[-tr(M£xV4 + u-'))]du 



= (£7r)"^("-")/2/C(«_„)/2(|x'a;). 
This gives ()4.1H) . Application of the Fubini theorem in this argument 



is justified because both integrals in ()4.11|1 converge absolutely. Indeed, 
for the left-hand side the absolute convergence is obvious, and for the 
right-hand side it follows from (J4.14J) . 

(ii) The case 2m > n + 2. In this case the interval ()4.12j) is void. 
To circumvent this difficulty, we replace f{x) in ()4.15p by |x|^A^/(x) 
and proceed as above, assuming k large enough and Re a > m — 1. 
Formally we obtain 

/^'[\y?::{:Ff){y)]\eIm + y'y\-'^"dy 

P . .2) J l^lm^''/(^)^(— n)/2 (|a;'x) dx. 

The equality ()4.16|) will become meaningful if we justify application of 
the Fubini theorem and specify a suitable interval for Re a. Clearly, 
the left-hand side of ()4.16p absolutely converges for all a G C, and 
one should take care of the right-hand side only. By Lemma 12.41 (ii). 



(4.16) nm/2 m(n-a)/2 
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the integral on the right-hand side absolutely converges for Re a > 
m — 1 — 2k provided 2k > 2m — 2 — n. Indeed, for small \x\m we have 

|x|^^AV(x)/C(,„„)/2 (|x'x) =0(|x|^-"), 

where 

{2k + n, if Re a > n + m — 1, 

2k + n + 1 — m — Eq, V^q >0, iin — 7n + l<Rea<n + m + l, 
2k + Re a, if Re a < n — m+ 1. 

If Re a > m — l — 2k and Eq is small enough {eq < 2k — 2m + n + 2) then 
A > r/i — 1 and the desired convergence follows. Thus, (J4.16J) is justified 
for Re a > m — 1. Since both sides of (j4.16|) are analytic functions of 
a in a larger domain Re a > m — 1 — 2k containing a non-void strip 

(4.17) m-l-2k < Rea <n-m+l, 

()4.16|) also holds in this strip, and one can utilize ()4.13|) and ()4.14|) in 
order to pass to the limit as £ — i> 0. This gives 

(4.18) Z{A%y\^^{Tf){y)l -a) = c^Z{A'f, a + 2k-n), 

(4.19) c, = vr-/22™("-")r„(!i^)/r„(|). 

To transform ()4.18p . we use the equality 

(4.20) A'lxC'- = B,{a)\xC-\ 

m— 1 fc— 1 

Bk{a) = Yl Y[{a -t + 2j){a-n + 2 + 2j+t) = Bk{n -a- 2k), 

i=0 j=0 

see ()3.7p . Then the right-hand side of ()4.18p becomes CaBk{a)Z{f, a — 
n). For the left-hand side we obtain (set ipiy) = \y\'^{J-'f){y), a = 
n- (3-2k) 

Z{A%\'J:{J^f){y)],-a) = Z{A'^,f3 + 2k-n) 

= B,{(3)Z{^,f3-n) 

= Bk{n-a-2k)Z{J^f,-a) 

= Bk{a)Z{J^f,-a). 

Finally, KT^ reads 

Z{T f , —a) = CaZ{f, a — n), m. — 1 — 2k<Rea<n — m, + l, 

and the desired equality ()4.10|) follows by analytic continuation. D 
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4.3. Normalized zeta distributions of integral order. It is con- 
venient to introduce a special notation for the normalized zeta integral 
Z{f, a — n)/Vm{a/2) which is an entire function of a. We denote 



(4.21) Ux) 



\x\ 



T^{a/2y 



iCaJ) 



1 



a.c. 



TUa/2) 



f{x)\x 






Tin 



where "a.c." abbreviates analytic continuation. We call C^ a normalized 
zeta distribution of order a. 

In view of forthcoming applications in Section 6, normalized zeta 
distributions of integral order deserve special treatment. A striking 
feature of the distribution (a is that ioi a = k, k = 0,1,2, . . .m — 1, 
(outside of the domain of absolute convergence !) it is a positive mea- 
sure. This measure is supported by a lower- dimensional manifold (in 
the rank-one case m = 1 we have only one point a = corresponding 
to the delta function at the origin). 

Definition 4.4. The set 

(4.22) W = {0,1,2, ... ,m - 1} U {a : Rea > m - 1} 

will be called a Wallah set of the normalized zeta distribution (a- 



This notion was introduced in JFK] for Riesz distributions associated 
to symmetric cones. Below we obtain explicit representations of (a for 
integral values of a (specifically, for < a < n) in the Wallah set 



Theorem 4.5. Let f G S{Tln,m)- For a = k, k = 1,2 

_(7i-fe)m/2 



(4.23) 



(a, /) 



TUn/2) 



d'j 



/ 7 



du. 



SO{n) OT,_„ 

Furthermore, in the case a = we have 

„nm/2 



(4.24) 



(Co,/) 



Tmin/2) 



/(O). 



Proof. STEP 1. Let first A; > m, — 1 . In polar coordinates we have 
Z{f,k-n) = [ f{x)\xt''dx 



dj. 



mn,m 








2~'^an,m 


r'/'-'dr [ f 


( 

7 


' ^1/2 " 




J 


J 

SOin) 


V 
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(3 G SO{k), then integrate in 



Now we replace 7 by 7 „ ^ 

I U In-k 

(3 G SO{k), and replace the integration over SO{k) by that over Vk^m- 
We get 



Z{f,k-n) 



2-"^a„ 



CTfc,; 



dl / Irl'^/^-d^r / /(7 

SO{n) Vm Vfc, 

(set u; = vr^^"^ G OTfc,m) 







CTr. 



C^kA 



rni 



k,m 



du / / ( 7 

50(n) 







d'y. 



dv 



This coincides with ()4.23j) . 

STEP 2. Our next task is to prove that analytic continuation of 
(Co;/) cit the point a = k {< m — 1) has the form ()4.23|) . We first 
note that for a = 0, ()4.24|) is an immediate consequence of ()4.10j) . Let 
k > 0. We split X G Mn^m in two blocks x = [y; b] where y G Mn^k and 
b G Mn.m-k- Then for Re a > m — 1, 



{(a, f) 



r™(«/2) 



dy 



fi[y;b]) 



y'y y'b 

b'y b'b 



{a-n)/2 



db 



M„, m„ 



where 






denotes the determinant of the respective matrix 

1/2 






By passing to polar coordinates (see Lemma f2 .61) y = tr^", f G ^, 
r E Vk, "^e have 



(C, /) 



Tmia/2) 



X 



fi[vr'/';b]) 



dv / |r|("-^-i)/2^r 



r r^/'^v'b 

Vvr^'"^ b'b 



(a-n)/2 



rf6 



2~ cr^t 



r™(a/2) 



^7 / Irl^^-'^-i^/^^r 



X 



SOCn) -Pfe 



A([V^/';&]) 



»i„ 



6'AorV2 5'5 



r^/^A^fe 



(a-n)/2 



rffe. 
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Here 


r _ H 




Ao = 


4 



e 14,fc, /^(x) = /(7a;). 


We write 






b = 


&2 


&1 e ^k,m-k, h e Tln-k,m-k- 


Since X'^b 


= bi, then 


(C.,/) = 




X 




^1/2 5^ - 

62 




b[r^/^ b'^bi + b'^b2 


Note that 






r 

.b'l 


rV26^ 
ri/2 b[bi + b'^b2 


= 


" r " 
ft'rV2 r , 

"1' J-m-k 




'Ik r-y% 
_ 6^62 


and 


det 


_ 6'iri/2 b[b, + 6^62 . 


= det(r)det(6262), 


see, e.| 


?., |Mnl 


p 


577]. Th 


erefo 


re, 













(a-n)/2 



(ifeo. 



(4.25) (C,,/) = c« y d^ j \4^-^-^y^dr 

where 

2-^a„,fcr„_fe((a-A;)/2) 



ipa-k{l,r,bi)dbi 



T^{a/2) 



ipa-kil,r,bi) 



r^_fc((o;-fc)/2) 



A 



ri/2 5^ 
62 



X |6'2&2|^""''^/'"^""^^/'c^69. 



'^''^■n — k.m — k 

'2- 



The last expression represents the normahzed zeta distribution of order 
a — k in the 62-variable. Owing to (J4.24J) . analytic continuation of (J4.25J) 
at a = A; reads 



(a,/) = Cfc / rf7 / \r\-'/'d: 






^1/2 ^^ 





dbi, 
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Cfc = [vT*^" '^^^'" ^"^^"^ /'^m-k{{n — k)/2)] lim Ca- To transform this expres- 



(3 

In-k 



sion, we replace 7 by 7 
This gives 

(a,/) = Ckj\r\-"^dr j dp 



Vk 







, P G SO{k), and integrate in /3. 
(i7 



X / A 

50(n) 

(set C = Pbi, V = 6|rp^ and use Lemma EEI) 



<^k± 



df] 



^k,k M,, 



^ , 


^a( 

\ 


'v C" 



SOin) 







duj, 
J 





d'y 



= c / ci7 / /(7 

SO(n) 9Jtfe,,„ 
C = :: TT^TTT lim — 



ak,k'rrn-k{{n - k)/2) a^k r„(a/2) 
(here we used formulae ()2.8|) and ()2.3H) ). 

The following formulas are consequences of ()4.2Hj) 

Corollary 4.6. For all k = 1, 2, . . . , n, 

(4.26) 



Tm{n/2) 



iCk, f) =ci dv / f{vuj)duj, 

(4.27) ci = 2-^- 7r("™-'='™-"^)/2 rfe(r2/2)/r„(n/2). 
Moreover, if k = 1,2, ... ,m — 1, then 

(4.28) (a,/) = c, I du I fiyu'M'^-^dy, 
and 



(4.29) (a,/) = C2 



I m-m 

|y|fc 



/([l/;i/2;])dz, 



n 



(4.30) C2 = vr(— ^)("/2-'=) /r,(fc/2) r„_fe((n - k)/2). 
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Proof. From ()4.23|1 we have 

r / r r, 1 

e Vn,k 



Ik 





^{n-k)m/2 f f ( 

{a J) = jrj^ j du j /(7Ao-)rf7 ^Ao = 

97tfc,m SO{n) 

^{n-k)m/2 r /• 

= r f /o^ / ^^ / fi'"^)dv 

an.ki^m{n/2) J J 

which coincides with (j4.26p . To prove (j4.28p . we pass to polar coordi- 
nates in ()4.26p by setting to' = ur^^"^, u E Vm,k, ^ E Vk- This gives 

[Q^J) = 2-^ci f dv f Irl^'^-^-^^^^dr f f{vr^^^u')du 

V„h Vk Vm,k 



ci J du J fiyn'M'^-'^dy. 



To prove ()4.29p . we represent u in ()4.26|) in the block form u = [rj] Q, 7] E 
Tlk^k, C ^ ^k,m-k, and change the variable ( = rjz. This gives 

{CkJ) = c, j \r^r-''dr^ j dz J f{v[v;vz])dv. 

Using Lemma f2 ■ 61 repeatedlv. and changing variables, we obtain 
{(k,f) = 2~'c,ak,kj\4"'-^-'^"dr j dz J f{v[r'/\r'/h])dv 

Vk 9jifc,„_fc yn,k 

= ciak,k / yr^ / f{[y;yz])dz. 

J \y\k J 

By dOZI), (1^^ and (EH), this coincides with (jOH). □ 

The representation ()4.29|) was obtained in |Shj and |Khj in a different 
way. An idea of ()4.28|) is due to E. Ournucheva. 

Remark 4.7. One can also write (Cfc, /) as 



(4.31) (Cfc,/)= j f{x)dUk{x), fES{mn,m), 

where Pk is a positive locally finite measure defined by 

(4.32) (z/fc,v5) = Ci dv ip{vuj)duj, V' e Cc(9Jl„,m), 

Vn,k Tlk,m 
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Cc{^n,m) being the space of compactly supported continuous functions 
on Tln^m', cf. (|4.26|) . In order to characterize the support of z/^, we 
denote 



(4.33) 



Tln,m{k) = {x : X e Mn,m, rank(x) = k}, 



(4.34) Tln,m{k) = [j Mn,m{J) (the closure of OT„,^(A;)). 



Lemma 4.8. The following statements hold. 
(i) suppz/fe = mn,m{k). 



h 




under the 



(ii) The manifold Tln,m{k) is an orbit of Ck - 

group of transformations 

X ^ gixg2, giEGL^UjE.), g2EGL{m,M). 

(iii) The manifold Mn^m{k) is a collection of all matrices x G Tln^m of 
the form 



(4.35) 

or 
(4.36) 



X = 7 



UJ 





X = vu, 



7 G SO{n), UJ G Tlk,: 



V G K,fc, ^ e DJik,; 



Proof, (i) Let us consider ()4.32|1 . Since rank(fco') < k, then {vkiV) = 
for all if G Cc{^n,m) supported away from dRn,m{k). This means that 
suppz/fe = m„,^(fc). 

(ii) Let us show that each x G Mn^m{k) is represented in the form 
X = giGkg2 for some gi G GL(?t,,R) and (72 ^ GL{m,M.). By Lemma 
12.51 each matrix x can be written as x = up, where u G \4,m, and p = 
is a positive semi-definite m x m matrix of rank k. By taking 



/^Al/2 : 



X X 



into account that p = g'2 
I, 



Ik 




M = 7 







(72 for some (72 G GL{m, M), and 

Km 

for some 7 G 0(n), we obtain x = giekg2 with 



^1=7 



^2 
/„ 



G GL{n,'. 



(iii) Clearly, each matrix of the form (j4.35|) or (|4.36p has rank < k. 



Conversely, if rank(x) < k then, as above, x 



up 



7 



P 




where 
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7 G 0{n) and p = (x'x)^/^ is a positive semi-definite m x m matrix of 
rank < k. The latter can be written as 

P = 9W, geO{m), A = diag(Ai,...,Afc,0, ...,0), 

and therefore, 



X = 7 



where 71 = 7 
done. Ifdet(7i^ 





9 
/. 



9 



7 



9 
/„ 



A 




9 =li 



ijj 




and uj e aJlfc,m. If det(7i^ 
1, one should replace 71 by 71 e, e = 



1, we are 



-1 

_ In-l _ 

and change uj appropriately. The representation ()4.36p follows from 



Corollary 4.9. The integral d^. Ji| ) can he written as 



(4.37) (a,/)= j f{x)duk{x)= J f{x)duk{x). 

vsink{x)<k rank(a;)=fc 

Indeed, the first equality follows from Lemma 14.81 (i). The second 
one is clear from the observation that if rank(x) < A; — 1 then by ()4.36|) . 
X = vu, V e Ki,fc-i! ^ £ 9^fc-i,m- The set of all such pairs {v,uj) has 
measure in Vn,k x ^k,m- 

5. Convolutions with zeta distributions and Riesz 

potentials 

5.1. Definitions. For a belonging to the Wallah set ()4.22|) . let us con- 
sider the convolution operator (a* defined by 



(5.1) 



iCa*f)ix) 



Tm{a/2) 



f{x-y)\yC--dy 



Mr, 



ii Rea > m — 1, and 



(5.2) 



{Ck* f)ix) =ci dv f{x-vuj)duj, 



if a = /c, /c = 1, 2, . . . , n; cf. Corollarv l4.6l Note that for m—1 < k < n, 
both representations are applicable to (k* f- 
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Another important normalization of the zeta distribution and the 
corresponding convolutions (j5.ip and (j5.2j) leads to the Riesz distribu- 
tions and Riesz potentials. We observe that the functional equation 
()4.10|) for the zeta function can be written in the form 

(5.3) ^r^^(/' « - ^) = (27r)-"-Z(^/, -a) , 

(5.4) 7n,m(a) = —f—r, n i^^ , " 7^ ra-m+1, n-m+2, .... 

r„((n-a)/2) 

Now we have excluded the values a = n — m + 1, n — m + 2, . . ., 
because these are poles of the gamma function Tm{{n — a)/ 2) sitting 
in the numerator in ()5.4j) (we recall that m > 2). The corresponding 
Riesz distribution ha is defined by 

1 1 /" _ 

(5.5) (haj) = --Z{f,a-n) = a.c. / |x|^ "/(a;)rfx, 

where / e S{DJln,m)- For Rea > m — 1, the distribution ha is regular 
and agrees with the ordinary function ha{x) = \x\^"'/'yn,m{o:). 
The Riesz potential of a function / G S{Mn,m) is defined by 

(5.6) {rf){x) = {ha J.) = -^--Z{f,,a-n), /.(■) = f{x-). 

For Re a>m~l,ay^n — m + 1, n — m + 2,..., ()5.6p is represented 
in the classical form by the absolutely convergent integral 

(5.7) {rf){a:) = -\- [ fix - y)\y\'^--dy. 

TO 

This integral operator is well known in the rank-one case m = 1. 

Riesz potentials of integral order deserve special mentioning. The 
following representations are inherited from those for the normalized 
zeta function; see Theorem 14.51 

Theorem 5.1. Let f G S{DJln,m)- Suppose that a = k is a positive 
integer. If k ^ n — m + 1, n — m + 2, . . ., then 



(5.8) il'f)ix) = 71 y ci7 y /(a;-7 

50(n) mk,m 

(5.9) = 72 / di; / f{x — vuj)duj, 



duj. 
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where 

' n — k\ ,„ /n 



(5.10) 71 = 2-'=-vr-^-/2r„(^)/r^( 

(5.11) 72 = 2-'=(""+^) 7r~'=(™+")/2 r^ 



2 

n — r/i^ 



The constant 72 above was evaluated by making use of ()2.9j) . 
The following theorem resumes basic properties of Riesz distributions 
and Riesz potentials. 

Theorem 5.2. Let f E S{Tln,m), a G C, a ^ n—m+1, n—m+2, ... 
(i) The Fourier transform of the Riesz distribution ha is evaluated by 
the formula {J-'ha){y) = |y|m"; ^^^ precise meaning of which is 

(5.12) {ha, f) = (27r)-"™(|y|„", (J'fM) = (27r)-"-Z(^/, -a). 
(ii) If k = 0,1, ... , and A is the Cayley- Laplace operator, then 

(5.13) i-ir''A'^ha+2k = ha, I.e. {-ir\ha+2k,A^f) = {ha,f), 

and, therefore, 

(5.14) {I-'^f){x) = (-1)'"^AV)(x). 
In particular, 

(5.15) {I'f){x) = f{x). 

Proof, (i) follows immediately from the definition ()5.5p and the func- 
tional equation ()5.3I ). To prove ()5.13|) . for sufficiently large a we have 

^ i>'a+2k[X — -, — — — ZA \X\ 

7„,„(a + 2k) 



\m 



Bkia) ^u. I 



7n,m(tt + 2/c) 
Cha{x), 



where by ^^ and (ETjl . 

Efc(a)7„,^(a) ^ Efc(a)r^(ct/2)r^((n - a)/2 - k) 
7„,„(a + 2A;) 4-'=r„(a/2 + A;)r„((n - a)/2) 



1. 



For all admissible a G C, ()5.13p follows by analytic continuation. The 
equality ()5.14|) is a consequence of (j5.12|l and ()5.13|) . Indeed, by ()5.6|1 . 



ir''f)ix) 
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= ih^2kJ.)^i-ir\ho, AV.) 




^ (-l)'"'^(27r) "™Z(J^(AVa=), 0) 




= (-1)™'=(AV.)(0) = (-i)™^aV)(x). 






D 



5.2. Riesz potentials and heat kernels. It is convenient to study 
Riesz potentials by making use of the heat kernels and the correspond- 
ing Gauss- Weierstrass integrals. In the rank-one case m = 1 this ap- 
proach is described in |Rulj . Section 16. In the higher rank case it 
was implicitly indicated in [CT| and |FKj . The key idea is to repre- 
sent the Riesz potential as a lower- dimensional fractional integral of 
the corresponding Gauss- Weierstrass integral which is easy to handle. 

Definition 5.3. For x G ^n,mi n > m, and t G Vm, we define the 
generalized) heat kernel ht{x) by the formula 

5.16) ht{x) = (47r)-"'^/2|^|-"/2exp(-tr(t-^x'x)/4) 

where |t| = det(t). The corresponding Gauss- Weierstrass integral 
Wtf){x) of a function /(x) on DJln,m is defined by 

5.17) {WJ){x)= j h,{x-y)fiy)dy= J hjjy)fix~yt'^')dy. 

Lemma 5.4. 

i) For each t G Vm, 

5.18) / ht{x)dx = l. 

ii) The Fourier transform of ht{x) has the form 

5.19) {J^ht){y)=exp{-tT{ty'y), 
which implies the semi-group property 

5.20) ht*K = ht+r, t,T e Vm- 
iii) /// G LP(2n„^m), l<p<oo, then 

5.21) ||m/||p<||/||p, WtW,f = Wt+rf. 
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and 

(5.22) \\m{Wtf){x) = f{x) 

in the L^-norm. If f is a continuous function vanishing at infinity, 
then \5.2^) holds in the snp-norm. 

Proof. To prove (i), we pass to polar coordinates. Owing to (|2.5p and 
fl2.Hl|l . we obtain 

hAx)dx = , , "",'"; ,, / exp(-tr(t-V)/4) WY'l^-'^dr 

_ 2 CTn^m J- Tn(^/2j i.-l ;^|-n/2 _ i 
~ (47]-)"'»/2 |i|n/2 I*- / I ~ • 

The formula ()5.19|) is well known (see, e.g., [Herz]), and the proof 
is elementary. Indeed, by changing variable x -^ 2xt^/^ and setting 
?7 = 2yt^''^, we have 

(J^/it)(y) =7r""™/^ / exp(-tr(ta;'a;)) exp(ztr(r7'x)) dx. 

This splits into a product of the one-dimensional Fourier transforms of 
Gaussian functions. The statement (iii) follows from ()5.18|) and ()5.2U|) . 
Indeed, the relations in ()5.21|) are clear. Furthermore, 

{WJ){x) - fix) = J hjJy)[f{x-yt'/')-f{x)]dy. 

Hence, by the generalized Minkowski inequality, 

\mf-f\\,< I hjJy)\\f{.-yt'/')-f{-)\\,dy. 

If t tends to the 0-matrix, then yt^^'^ ^ in R"™. Since the integrand 
above does not exceed 2\\f\\phi^{y), we can pass to the limit under 
the sign of integration, and the desired result follows. For continuous 
functions vanishing at infinity, the argument is similar. D 

Remark 5.5. A challenging open problem is whether (|5.22j) holds for 
almost all x G Mn,m- This fact is well known in the case m = 1 [St2], 
[SW]. It follows from the estimate 

(5.23) sup \{Wtf){x)\<{M*f){x) 
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where {M*f){x) is the Hardy-Littlewood maximal function. It would 
be desirable to extend this theory to the matrix case when the positive 
parameter t is replaced by a positive definite matrix. 

Let us consider the Garding-Gindikin fractional integrals on Vm de- 
fined by 

oo 

(5.24) {I'_gm = f^J 9{r)\T-t-'dT, 

t 

where d = (m + l)/2, Re A > d — 1, and integration is performed over 
all r G Vm so that t — t E Vm ■ The following theorem establishes con- 
nection between the Riesz potentials, the Gauss- Weierstrass integrals, 
and the Garding-Gindikin fractional integrals. 

Theorem 5.6. Let m — 1 <Rea<n — m + l, d = (r7i + l)/2. Then 

(5.25) (r/)(x) = -^-— [ \tr/'-\WJ){x)dt, 

Tm{a/2) J 

Vm 

(5.26) WArf]{x) = d\W^.)f){x)]{t) 

provided that integrals on either side of the corresponding equality exist 
in the Lehesgue sense. 

Proof. The right-hand side of ()5.25j] transforms as follows: 

r.h.s. = \^^-- / |t|(--")/2-d^t / /(y)exp(-tr(ri^))rfy 
Tm{al2) J J 

where z = {x — yy{x — y)/4:. Now we change the order of integration 
and replace t~^ by t. Since 

\{n-a)/2-d ^^^(_fr(+'.\\ rl+ = V f^^^ \ I ^\{a-n)/2 



|t|^"-°^/'-^exp(-tr(te))rft = Tm [ 1 l^r"""^/', \z\ ^ 0, 

Vm 

(I5.25|l follows. The validity of ()5.26j) is a simple consequence of the 
semi-group property of the Gauss- Weierstrass integral. Indeed, 

Wt[IV]{x) = V f\rf^-\WtWrf){x)dr 



Tm{a/2) 

V, 

ja/2 



\T 



'''^-\Wt-,rf){x)dT 



m^-)f){^)m. 
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D 

Theorem 15.61 has a number of remarkable consequences. Firstly, it 
enables us to invert the Riesz potential by inverting I_ and applying 
the approximation property limWtf = f. We shall study this question 

in subsequent publications. Secondly, Theorem 15.61 provides a simple 
proof of the semigroup property of the Riesz potentials under mild 
assumptions for /. 

Corollary 5.7. Let 

(5.27) Rea > m-l, Re/3>m-l, Re{a + /3) < n - m + 1. 
Then 

(5.28) (/°/^/)(a:) = (/°+^/)(a;) 
provided that the integral {1°"^^ f){x) absolutely converges. 

Proof. Applying Theorem 15.61 and changing the order of integration, 
we obtain 



ri^f = } f\tr/'-'wti^fdt 

Tm{a/2) J 

itr/2"'(//V(.)/)(t)dt 



Tm{a/2) 
r, 

1 



T 



r^(a/2) r^(/3/2) 

Vm 

1 ^V|("+^)/2-'^Vr,/t/r = r+^/. 



r™((a + /?)/2) 



U 

Remark 5.8. For m = 1, the conditions ()5.27|) have a well-known form 

i?ea>0, i?e/3>0, Re{a + [3) <n. 

Note that in the higher rank case, ()5.27p is possible only if Sm < 
ra + 3. It is also worth noting that the "usual" way to prove the semi- 
group property (j5.28p for "rough" functions / via the Fourier transform 
formula and evaluation of the corresponding beta integral (cf. [SKM] , 
[St2]) becomes much more difficult in the higher rank case. 

Remark 5.9. Since the "symbol" (i.e., the Fourier transform of the 
kernel) of the Riesz potential /"/ is \y\^, then, for sufficiently good 
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functions /, ()5.28|1 holds for arbitrary complex ot and /3. We will be 
concerned with this topic in Section 6. 

5.3. L^-convolutions. Another useful application of Theorem 15.61 is 
the following. By Lemma f4. 21 and Theorem 14.51 convolutions with zeta 
distributions and Riesz potentials are well defined on test functions 
belonging to the Schwartz space. Are they meaningful for / G L^ ? Let 
us study this question. 

Theorem 5.10. For j G Ui^n^m), the Riesz potential (/"/)(x) abso- 
lutely converges almost everywhere on Mn,m provided 

Ti 

(5.29) Rea>m-l, l<p< 



Rea + m — 1 

Proof. Note that (jK^^ is possible if m < [(n + l)/2] where [(n + l)/2] 
is the integral part of (n + l)/2. The conditions (|5.29|) are well known 
i{ m = 1 [Stl] when they have the form 1 < p < n/Rea. To prove 
our theorem, it suffices to consider / > and justify the following 
inequality 

(5.30) 1= I exp(-tr(x'x))(/"/)(x)c^x< c||/||p, c = const. 
TO 

Regarding / as the Gauss- Weierstrass integral {Wijyl'^ f]){S^) owing to 
(I5.26|) . we have 






In 



r - /„^^/'"'^ |rr"/2 ^7- / exp(-tr(r-^x'x)/4) f{x) dx, 
d = {m + l)/2. By Holder's inequality, / < A||/||p where 

oo 

AP' = [ \t- Iml'"^^-'^ kr"^^( / exp(-p'tr(r-^a;'x)/4) dxY^^ dr, 
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1/p + 1/p' = 1. This integral can be easily estimated. Indeed, by 
setting X = yr^^"^, dx = \T\"'^'^dy, we obtain 



AP' 



|^_X^|«/2-dU|-n/2p^^ fr-l 



1/ _ ^|"/2-d |^|(n/p-a^)/2-(i^j. 



The last integral is finite if p obeys ()5.29|) . 



D 



More precise information can be obtained for convolutions with zeta 
distributions (and therefore, for Riesz potentials) of integral order. For 



A; = 1, 2, ... n — 1, we denote 



(5.31) Vn-k-- 





^n—k 



^^n,n-kj -rrig"-'= — 'i^ra-fc'^n-fc " 





In-k 



(5.32) 



SO{n) 



Lemma 5.11. Let n > m > 1; k = 1,2, . . . ,n. For any X > k + m — 1, 



(5.33) 



\i(k*f){x)\ 



dx = C\ 



fix) 



|/„ + a;'Prffin-fcx|(^-^)/2 



dx. 



Mn 



Tin 



(5.34) 



cx 



7r""^/^r^((A-A;)/2) 
r^(n/2)r„(A/2) 
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Proof. Suppose first that k < n, and denote the left-hand side of ()5.3Hj) 
by /(/). By (^3), 



y^.k Tik, 



m^,r. 



(set V = -fvo, vo = 
c.a . f duj f f 


h 



, z 


= -fy, 7 G SO{n)) 
dx 


Ci(rn,k / db f i 


VqUJ 

b 


]d- [ ^" 


J ^ J |/„ + 6'6 + a'a|V2 



For A > k + m — 1, the last integral can be explicitly evaluated by the 
formula 



(5.35) 



da 



\q + a'a\^/'^ 



7rW2r^((A-A;)/2) ,^,(,_,^/, 



Tl. 



r^(A/2) 



(1\ 



geP„ 



(see (A. 3)). This gives ()5.33p . For k = n, the proof is simpler and 
based on ()5.35p . In this case Prjjn-fc should be replaced by the zero 
matrix. D 

Corollary 5.12. If n > m > 1, k = 1,2, ... ,n, and X > k + m — 1, 
then 

\iCk*f){x)\ 



(5.36) 



\Im + X'xl-^/^ 



dx < ca||/||i. 



Mr,. 



If k = n and / > 0, then i5. 3b]) is a strict equality. 

Let us extend ()5.36|) to / G L^ with p > 1. Now we have to im- 
pose extra restrictions because of the projection operator PrK7i-fc on 
the right-hand side of ()5.33|) . 

Theorem 5.13. Let n > m; k = 1,2, . . . ,n — m. If f E V and 



\> k + max I m — 1 



then 
(5.37) 



n + m — 1 
pi 



\Im + X'X\^I^ 



P p 



c = c{X,p, n, k, m). 



aji„ 
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provided 

(5.38) 1<P<-, 7- 

^ ^ ~ k+m- 1 

Proof. Note that ()5.38p agrees with ()5.29p in Theorem 15.101 It suffices 
to prove the statement for non- negative radial functions f{x) = fo{x'x). 
We remind the notation d = {m + l)/2. Denote by /(/) the left-hand 
side of ()5.37p and make use of ()5.33|) . Splitting the integral in the 
right-hand side of (15.331) in fnyy x /mr ; and passing in fm^ 

to polar coordinates, we have 

/\^\{n-k)/2-d r 

^^-—^^—^dr J fo{uj'uj + r)du 

OO 



^^ / /o(s)^^ ■ / l(X-k)/2 ^-^ 



m 



k.m 



foi-V^-'^ / TT — \ / \(x-k)/2 ^^ 

Vm U)'u)<S 

(set LU = rjs^'^, 7] G Mk^mi duj = \s\''''^ drj) 

r \T _ „'„|("-fc)/2-rf 

Vm ri'r]<Im 

By taking into account that 

^ i/p 
(5.39) ll/llp^ I j \Us)y\sT"-'ds 

owing to the Holder and the generalized Minkowski inequality, we ob- 
tain /(/) < c||/||p where 

ri'v<Im 



/\g\n/2-d 
|/^ + gi/2^gi/2|M/2 ^^> /i=(A-A;)p', r = I^-r]'ri. 

Pm 
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Since \Im + s^^'^rs^^'^] = \I„i + r^^'^sr^^'^], by changing variable s = 
^-1/2^^-1/2^ we have 

The last integral is finite for fi>n + 'm — 1ot{X — k)p' > n + m — 1 
(moreover, it can be explicitly evaluated by (A. 2)). Thus for A > 
k + {n + m — l)/p' we have 



r}'q<Irr 



ir _H("/P-'=)/2-'^|r|-i/2dr 



< oo 



provided n/p — k > m — 1, i.e., 1 < p < n/{k + m — 1). This completes 
the proof. D 

Remark 5.14. We do not know whether the conditions (J5.38J) and (J5.29J) 
are sharp. Moreover, it would be interesting to study boundedness 
properties of the Riesz potential operator J" on functions / G L^(97t„^m) 
in different norms. 



6. Radon transforms 

6.1. Definitions and basic properties. Let k,n, and m be positive 
integers, < k < n, Vn^n~k be the Stiefel manifold of orthonormal 
{n — /i;)-frames in R". For C, G Vn,n-k, and t G Mn-k,m, the linear 
manifold 

(6.1) r = r(e, t) = {xe mn,m ■■ i'x = t} 

will be called a matrix k-plane in Tln,m- We denote by T the variety 
of all such planes. For m = 1, 1 is an affine Grassmann manifold of 
/c-dimensional planes M."'. The matrix /c-plane Radon transform /(r) 
of a function f{x) on DJln,m assigns to / a collection of integrals of / 
over all matrix planes r G T. Namely, 

/(r) = / fix), TET. 
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The dual Radon transform <p{x) is a mean value of a function <f{T) on 
1 over all matrix planes r through x: 



(f{x) = / !f{T), X e Mn^m- 



t3x 



Precise meaning of these integrals is the following. Let r be the plane 
G Vn,n-k, and g^ G SO{n) be a rotation satisfying 



(EID,eo 



fi-^^o = ^- Then 



(6.2) 

and 

(6.3) ^{x) 



/(r)^/(e,t)= J f{g^ 






(ici;, 



^n,n—k 



^{^,ex)d^= / ^(7eo,e;7'a:)ci7. 



50(n) 



Lemma 6.1. 

(i) The duality relation 



(6.4) 



f{x)(p{x)dx 



^n,n—k 



d^ / Vi^,t)fi^,t)dt 



m^. 



Vn,n-k VKn-k,m. 



holds provided that either side of \d.4\} is finite for f and (p replaced by 
I/I and \ip\, respectively. 

(ii) If f & L^{DJln,m), then the Radon transform f{^,t) exists for all 
C ^ yn,n~k o-nd almost all t G Mn~k,m- Furthermore, 



(6.5) / fi^,t)dt= I fix)dx, VeGK,n-fc. 

Proof. By (j6.2p . the right-hand side of (j6.4j) has the form 



(6.6) 



(T, 



1- J d^ J ^{U)dt J f^g^ ""t )du; 

Vn,n-k Vjl„-k,m -Olk.m 



Changing variables x = g^ 



t 



we have 



^'x = (^e^o)'^c 



t 



eo 



Ijj 

t 



t. 
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Hence, by the Fubini theorem, (jfi.fij) reads 

/ c/^ / yD{^,^'x)f{x)dx = / ip{x)f{x)dx. 

^n,n—k J J J 

Vr, „-h DT? 97? 

The statement (ii) is a consequence of the Fubini theorem: 
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dt j f [g^ 

71— k, 771 ~'-' ^ fc , m 

f{g^x) dx 



t 



doj 



f{x) dx. 



Win, 



OJln, 



D 



The following properties of the Radon transform can be easily checked. 
Lemma 6.2. Suppose that the Radon transform 

fix) > /(^,t), X e mn,m, (^,t) e Vn,n-k X 9Jln-fc,m, 

exists (at least almost everywhere) . Then 

(6.7) f(^e\et) = fi^,t), \/eeO{n-k). 

U 9{x) = 7X/5 + y where -y G 0{n), [3 G 0{m), y G ^n,m, then 

(6.8) U^9)\i,t) = f{ii,tP + i'i'y)- 

In particular, if fy{x) = f{x + y), then 

(6.9) m,t) = f{^,ey + t). 

The equality ()(i.7|l is a matrix analog of the "evenness property" of 
the classical Radon transform, cf. [Hel]. 

It is known [ORl] that the Radon transform / ^ / is injective on 
the Schwartz space S = <S(27l„^m) if and only if A; < n — m. The 
classical problem is how to reconstruct a function / from the integrals 
/(r), T ^ 1. One of the ways to do this lies via Riesz potentials l''f of 
integral order. By Theorem 15. 11 for 1 < k < n — m and / G iS we have 



(6.10) 



(6.11) 



(/V)(a:)=7i j dl j f(x-j 







duj. 



7i 



2-.n.^-W2r„(^!i_^)/r^^^ 
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By Theorem I5.13[ this expression is well defined a.e. on Tln,m as 
an absolutely convergent integral for any f & L^ provided 1 < p < 
n/{k + 771 — 1). The following important statement establishes connec- 
tion between Riesz potentials and Radon transforms. It generalizes the 
classical result of B. Fuglede; see |Fuj . |Helj for m = 1. 

Theorem 6.3. Let 1 <k <77 — m. Then 

(6.12) 7i(/y(a;) = (/V)(x), 

provided the Riesz potential on the right-hand side absolutely converges, 
e.g., for f G L^ , 1 < p < n/{k + m — 1). 

Proof. Let f^{y) = f{x + y). Then ^^ yields /,(e, t) = f{^, i'x + t). 
Owing to ()6.3p and ()6.2p . by changing the order of integration, we have 

U)\^) = ^— / /(e,e'a:)rfe = ^— / m,o)d^ 

*n ,n~~k *n ,n — k 







duo. 



dl j / ( a; + 7 

SO{n) gjifc,,„ 

This coincides with 'j^'^{l''f)(x). D 

Corollary 6.4. If f E L^, I < p < n/{k + m — 1), then fi^r) is finite 
for almost all r G T. 

Remark 6.5. The condition 1 < p < n/{k + m — 1) in Corollarv 16 .41 can 
be replaced by the weaker one, namely, 

71 + m — 1 

see [0R2]. The proof of this fact relies on completely different ideas 
related to representation of the Radon transform of radial functions 
by the Garding-Gindikin fractional integral ()5.24|) . For m = 1, both 
conditions coincide and cannot be improved. We are not sure that for 
m > 1, (j6.13p is sufficient for the existence of the Riesz potential I'^f 
(in contrast to the case m = 1); cf. Remark 15. 141 

6.2. The inversion problem. Theorem l6.3l reduces the inversion prob- 
lem for the Radon transform to that for the Riesz potentials (as in the 
rank-one case [Hel], [Ru2]). Below we show how the unknown function 
/ can be recovered in the framework of the theory of distributions. 
Let us consider the Riesz distribution 

(6.14) ^h^j) = a.c.^-- [ \x\'^-y{x)dx, 
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where / G 5 = S{DJln,m)- From the Fourier transform formula 

(/.«,/) = (27r)--(lz/l™^(-^/)(l/)), 

it is evident that the Schwartz class S does not suit well enough because 
it is not invariant under multiplication by ||/|~". To get around this 
difficulty, we choose another space of test functions. Let \E' = \E'(97l„,„) 
be the collection of all functions ipi^y) G 5(9Jl„_m) vanishing on the 
manifold 

(6.15) V = {y : ye aH„,^, rank(y) <m} = {y : ye OT„,„, \y'y\ = 0} 

with all derivatives (the coincidence of both sets in (jfi.lSj) is clear be- 
cause rank(y) = rank(?/'y)). The manifold V^ is a cone in M""^ with 
vertex 0. The set ^ is a closed linear subspace of S. Therefore, it can 
be regarded as a linear topological space with the induced topology of 
S. Let $ = ^(DJln,m) be the Fourier image of \l/. Since the Fourier 
transform JF is an automorphism of S (i.e., a topological isomorphism 
of S onto itself), then $ is a closed linear subspace of S. Having been 
equipped with the induced topology of S, the space $ becomes a linear 
topological space isomorphic to \E' under the Fourier transform. 

The spaces $ and \E' were introduced by V.I. Semyanistyi [Se] in the 
rank-one case m = 1. They have proved to be very useful in integral 
geometry, fractional calculus, and the theory of function spaces. Fur- 
ther generalizations and applications can be found in [Li], [Rul], [Sa], 
[SKM]. 

In our case the following characterization of the space $ is a conse- 
quence of a more general result due to S.G. Samko [Sa]. 

Theorem 6.6. The Schwartz function (j){x) on Tln,m belongs to the 
space $ if and only if it is orthogonal to all polynomials p{x) on any 
hyperplane t in M"™ having the form r = {x : tr(a'x) = c}, a e V : 

(6.16) / p{x)(j){x)diJi{x) = 0, 



d^{x) being the induced Lebesgue measure on r. 

Note that for m = 1, the space $ consists of Schwartz functions 
which are orthogonal to all polynomials on M". 

We denote by $' the space of all linear continuous functionals (gener- 
alized functions) on $. It is clear that two iS'-distributions that coincide 
in the $'-sense, differ from each other by an arbitrary iS'-distribution 
with the Fourier transform supported by V . Since for any complex a, 
multiphcation by \y\'^ is an automorphism of \E', then, according to 
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the general theory [GSh2], /" is an automorphism of $, and we have 

(6.17) -^[/>](t/) = |y|-°-F[0](i/), e $. 

This gives 

(6.18) ri>^(j) = r+f^(f) Va,/3GC 
(cf. Remark l5.9|l . and 

(6.19) ^[r/](2/) = b|-"^[/](y) 

for all ^'-distributions /. The last formula implies the following 

Theorem 6.7. The function f E U'{mn,Tn)-, 1 < P < n/{k + m - 1), 
can he recovered from the Radon transform g = f in the sense of $'- 
distributions by the formula 



(6.20) 




(/,0)=7i 


(^,/-V), <^e<D, 


where 












(/-V)(x) 


= (^-'|y|^-^0)(x), 


7i being the 


I constant i6.ll]}. 




Proof. We have 








(/,0) 


= (27r) — (^[/],^M) 
= (2vr)-""^(l/-'=.F[/](y), ?/^.^[</>](|/)) 
= {2nr--{iT[l'f]){y), ytJ-my)) 
= (27r)-"-7i((-^[^])(y), l/™-^M(l/)) 



li{9,I~ 



D 



Remark 6.8. For A; even, the Riesz potential /*'/ can be inverted (in 
the sense of ^'-distributions) by repeated application of the Cayley- 
Laplace operator Am = det{d'd), d = (d/dxij). This operator agrees 
with multiplication by (— 1)™|?/|^ in the Fourier terms, and therefore, 
{-l)''"^A^l''f = / in the $'-sense. 

Remark 6.9. It would be desirable to obtain pointwise inversion for- 
mulas for I^f and / (not in the $'-sense). In the rank-one case such 
formulas can be found in [Rul] and [Ru2]. 
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7. Appendix 

The formulas in this section are not new. Since it is not so easy 
to find simple proofs of them in the literature, we present such proofs 
for convenience of the reader. We recall that Vm denotes the cone of 
positive definite m x m matrices, Vm is the closure of Vm, Bm is the 
beta function ()2.10|) . d = {m + l)/2. The following formulas hold: 



(Al) / |r|-^|r - sl^-^dr = |s|"-^E^(a,7 - a), 

s 

s G Vm, Rea > d — 1, Re {'j — a) > d — 1; 

oo 

iA.2) [ \I^+r\-^\r-s\''-''dr = \Im+sr^Bmia,^-a), 



s e Vm, Rea > d — 1, Re {^y — a) > d — 1; 
(A.S) J \b+y'yr^/^dy = ^^^^^^^^^^ 

heVm, ReX>k + m-l; 

(A4) J \b-y'y\(^-'^y^-'dy = "'"^^^^'^ ^^'^^ \h\'''-', 

{ye3Jtk,m- y'y<b} 

heVm, ReX> k + m-l. 

Proof. 

(A.l), (A. 2). By setting r = q~^, dr = \q\^''^^^dq, one can write the 

left side of (A.l) as 



and we are done. The equality (A. 2) follows from (A.l) if we replace s 

and r hy Im + s and /„ + r, respectively. 

(A. 3), (A. 4). By changing variable y -^ yb^^"^, we obtain 

\b + y'y\-'/'dy=\b\^''-'y'j„ 



an... 



/ \b-y'y\^'-'y'-'^dy=\b\'/'-'j„ 



{y&^k,m- y'y<b} 
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where 



J2 = I \I^-y'y\^'-'^/'-'dy. 



feeO^fc,™: y'y<im} 
Thus we have to show that 

7rW2r„((A_fc)/2) 



Jl = J2 



r™(A/2) 



The case k > m. We write both integrals in the polar coordinates 
according to Lemma f2. 61 For Ji we have 

- 9-™^ R (- ^~^ 

(the second equality holds by (A. 2) (with s = 0, a = k/2, 7 = A/2). 
Similarly, 

J2 = 2-^afc,^ /"|r|^/2-d|j^_^|(A-fc)/2-d^^ 

- 9-'"n- R fk \-k 

^ '-'k,m^i 



m, 2' 2 



Now the result follows by (ITTn|) and (IT!TT| . 

The case k < m. We replace j/ by y' and pass to the polar coordi- 
nates. This yields 



^1 = y \lm + yy'\-^'^di 



ty 

2-'' I dv I \Im + vqv'\-^''^\q\^"'-^-^^''^dq, 

Vm.,k Tk 

{\Im + vqv'\ = \Ik + q\) 
2-'a^,,J\q\^-'-'y'\h + qr'/'dq. 

Vk 
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By (A.2) (with s = 0, m = /c, 7 = A/2) and fl^ . 



k. 



Jl — 2 (TmhB, 



m 



m,k^k I n ' 



X — m 



Similarly, 












J2 = 


J 


[ 

■■ yy'<im} 


\Im 


— '. 


VI 


= 


2-' 1 


dv 


j 








Vm,k 


{gePfc: 


J 

vqv' 


</m} 






h 









2 ' 2 

7r'=™/2rfc((A-m)/2) 

7rW2r^((A-fc)/2) 
r™(A/2) 



2-V^,fc / 14 - g|(A-™-'=-l)/2|^|(™-A.-l)/2^^ 



-fc D /'"^ X — m 
2 am.kBk — , 



2 ' 2 



and we get the same. 
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